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1 Introduction and notation 

This article is motivated by the program of classifying actions of higher rank 
lattices in simple Lie groups on closed manifolds. More specifically, we are 
concerned here with actions of SX(n,Z) with n > 3, on closed oriented surfaces. 
A standard example of such an action is given by projectivizing the usual action 
of <SX(3, Z) on R 3 . Our objective here is to show that there are essentially no 
such actions which are symplectic or, what amounts to the same thing in this 
case, area preserving. 

R. Zimmer conjectured (see Conjecture 2 of [30]) that any C°° volume pre- 
serving action of a finite index subgroup of SL(n, Z) on a closed manifold with 
dimension less than n, factors through an action of a finite group. We prove 
this in the case that the dimension of the manifold is 2. L Polterovich [24] pre- 
viously provided a proof of this result for surfaces of genus at least one. Our 
aim was to extend this to the case of the sphere. As it happens our techniques, 
which are quite different from his, are equally applicable to any genus so we 
present the argument in full generality. 

Definition 1.1 A group Q is called almost simple if every normal subgroup is 
either finite or has finite index. 

The Margulis normal subgroups theorem (see Theorem IX.5.4 of [21] or 8.1.2 of 
[29]) asserts that an irreducible lattice in a semi-simple Lie group with R— rank 
> 2 is almost simple. In particular, any finite index subgroup of SL(n, Z) with 
n > 3 is almost simple. 

Suppose S is a closed oriented surface and lo is a smooth volume form. We will 
generally assume a fixed choice of u> and refer to a diffeomorphism F : S — > 
S which preserves to as an area preserving diffeomorphism. We denote the 
group of diffeomorphisms preserving u> by Diff a; (S l ) and its identity component 
by Diff w (S')o. Equivalently Diff w (S')o is the group of diffeomorphisms which 
preserve u> and are isotopic to id. 

A key ingredient (and perhaps limitation) of our approach to this problem is 
the fact that a finite index subgroup of SL[n, Z) with n > 3 always contains a 
subgroup isomorphic to the three-dimensional integer Heisenberg group. Recall 
that this is the group of upper triangular 3x3 integer matrices with all diagonal 
entries equal to 1 . Our main result is: 

Theorem 1.2 Suppose Q is an almost simple group containing a subgroup 
isomorphic to the three-dimensional integer Heisenberg group, e.g. any finite 
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index subgroup of SL(n, Z) with n > 3. If S is a closed oriented surface then 
every homomorphism (ft: Q — > Diff (i) (5) factors through a finite group. 

Given Q,S and <p as in Theorem 1.2, let <p' : Q — > MCG(S) be the homomor- 
phism to the mapping class group MCG(S) induced by 0. Since nilpotent 
subgroups of MCG(S) are virtually abelian (see [2]), the kernel Qq of 4>' has 
an infinite order element in the integer Heisenberg subgroup. Hence by al- 
most simplicity Go has finite index in Q . Moreover, Qq contains a finite index 
subgroup of the integer Heisenberg subgroup and hence contains a subgroup 
isomorphic to the three-dimensional integer Heisenberg group. Thus in proving 
Theorem 1.2 there is no loss in replacing Q with Qq. In other words, we may 
assume that the image of (j) is contained in T)iE UI (S)o- 

If n > 4, then any analytic action by S'L(n,Z) on a closed oriented surface 
S factors through a finite group. This was proved by Farb and Shalen [7] for 
S + T 2 and by Rebelo [25] for S = T 2 . (Farb and Shalen proved this for 
S = T 2 under the assumption that the action is area preserving.) Polterovich 
(see Corollary 1.1. D of [24]) proved that if n > 3, then any action by SL(n,7j) 
on a closed surface S other than S 2 and T 2 by area preserving diffeomorphisms 
factors through a finite group. All of these results can be stated in greater 
generality than we give here. There is also an analogous result of D. Witte 
([28]), which asserts that a homomorphism <j>: SL(n,Z) — > Homeo{S l ) must 
factor through a finite group if n > 3. 

We are grateful to Benson Farb for introducing us to the problem and for several 
very helpful converations. We are also grateful to David Fisher for suggestions 
that significantly streamlined and helped organize the paper. 



2 Hyperbolic structures and normal form 

Some of our proofs rely on mapping class group techniques that use hyperbolic 
geometry. In this section we establish notation and recall a result from [16]. 
For further details see, for example, [3]). 

Let S be a closed orientable surface. We will say that a connected open subset 
M of S has negative Euler characteristic if H\(M, R) is infinite dimensional 
or if M is of finite type and the usual definition of Euler characteristic has 
a negative value. If M has negative Euler characteristic then M supports a 
complete hyperbolic structure. 
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We use the Poincare disk model for the hyperbolic plane H . In this model, 
H is identified with the interior of the unit disk and geodesies are segments 
of Euclidean circles and straight lines that meet the boundary in right angles. 
A choice of complete hyperbolic structure on M provides an identification of 
the universal cover M of M with H . Under this identification covering trans- 
lations become isometries of H and geodesies in M lift to geodesies in H. 
The compactification of the interior of the unit disk by the unit circle induces 
a compactification of H by the 'circle at infinity' S^. Geodesies in H have 
unique endpoints on . Conversely, any pair of distinct points on Soo are the 
endpoints of a unique geodesic. 

Suppose that F: S — > S is an orientation preserving homeomorphism of a 
closed surface S and that M C S is an open connected F-invariant set with 
negative Euler characteristic. Equip M with a complete hyperbolic structure 
and let f = F\m ■ M — > M. We use the identification of H with M and write 
/ : H — > H for lifts of /: M — > M to the universal cover. A fundamental 
result of Nielsen theory is that every lift f:H^H extends uniquely to a 
homeomorphism (also called) / : H U — > H U . (A proof of this fact 
appears in Proposition 3.1 of [18]). If /: M — > M is isotopic to the identity 
then there is a unique lift /, called the identity lift, that commutes with all 
covering translations and whose extension over is the identity. 

Every covering translation T: H — > H extends to a homeomorphism (also 
called) T : H U — > iJ U S"^ . If 7 C M is a closed geodesic and 7 C i? is 
a lift to H then there is an extended covering translation T whose only fixed 
points T + and T~ are the endpoints of 7. If 7(7) is isotopic to 7 then there is 
a lift / : H U — > -ff U that fixes T 1 * 1 and commutes with T . The quotient 
space of H U {S^ \ T ± ) by the action of T is a closed annulus on which / 
induces a homeomorphism denoted / : A — > A . 

The following result is an immediate corollary of Theorem 1.2 and Lemma 
6.3 of [16]. If Fix(F) is finite, then it is just a special case of the Thurston 
classification theorem [27]. 

Theorem 2.1 Suppose that F: S — > S is a diSeomorphism of an orientable 
closed surface, that F is isotopic to the identity and that Fix(F) / 0. Then 
there is a finite set R of simple closed curves in 5\Fix(F) and a homeomorphism 
(j) isotopic to F rel Fix(F) such that: 

(1) (j) setwise fixes disjoint open annulus neighborhoods Aj C (S \ Fix(F)) of 
the elements jj G R. The restriction of <p to cZ(Aj) is a non-trivial Dehn 
twist of a closed annulus. 
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Let {Si} be the components of S \ UAj , let Xi = Fix(F) n Si, and if Xi is 
finite, let Mj = Si\Xi. 

(2) If Xi is infinite then (j)\s i = identity. 

(3) If Xi is finite, then Mi has negative Euler characteristic and 4>\m 1 is either 

pseudo-Anosov or the identity. 

We say that <p is a normal form for F and that R is the set of reducing curves 
for (/). If R has minimal cardinality among all sets of reducing curves for all 
normal forms for F , then we say that R is a minimal set of reducing curves. 

3 The proof of Theorem 1.2 

We denote by [G, H] the commutator G^H^GH. 

Proposition 3.1 If G,H: S — > S are diffeomorphisms that are isotopic to 
the identity and that commute with their commutator F = [G, H] then F is 
isotopic to the identity rel Fix(F) . 

Remark 3.2 In any group containing elements G and H, if F = [G,H] 
commutes with G and H then it is easy to see that [G k , H] = [G, H] k = [G, H k ] 
for any fc G Z. Hence F klk2 = [G kl , H k2 } for all k u k 2 . 

As an immediate consequence of this and Proposition 3.1 we have 

Corollary 3.3 Suppose that G,H : S — > S are diffeomorphisms that are iso- 
topic to the identity and that commute with their commutator F = [G, H] . 
Then for all n > 0, F n is isotopic to the identity rel Fix(F n ) . 

Before proving Proposition 3.1 we state and prove some required lemmas. De- 
note the closed annulus by A and its boundary components by <9oA and d\K. 
The universal cover A is identified with Ix [0,1]. 

Lemma 3.4 Assume that Ui : A — > A, i = 0, 1, are homeomorphisms that 
preserve do A and d±A, that Ui commutes with v = [ui, 112] and that iti\ A — > A 
are lifts of Ui . Then v = [u\, U2] hxes at least one point in both OqA and d\A. 
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Proof Let p\ : A — > R be projection onto the first coordinate and let T: A — > 
A be the indivisible covering translation T(x,y) = (x + l,y). We claim that 
|pi(£(x)) — < 8 for all x G <9A. 

Since the lift Ui commutes with T, v is independent of the choice of U{. We 
may therefore assume that \p\(ui(xi)) — pi{xi)\ < 1 for some Xi G <9oA and 
hence that \pi(ui(x)) — pi(x)\ < 2 for all x G doA. The claim for x G cfoA 
follows immediately. The analogous argument on <9iA completes the proof of 
the claim. 

The preceding argument holds for any n« that preserve the components of 
dA and in particular for all iterates uf . We conclude (Remark 3.2) that 
\pi{v N (x)) — pi(x)\ < 8 for all x G dA and all N. Since the restriction of 
v to each boundary component of A is an orientation preserving homeomor- 
phism of the line, these homeomorphisms both have points with a bounded 
forward orbit, and any such homeomorphisms of the line must have a fixed 
point. □ 

An isolated end E of an open set U C S has neighborhoods of the form 
N(E) = S 1 x [0, 1). The set fr(E) = cl s {N(E)) \ N(E) , called the frontier of 
E, is independent of the choice of N(E). We will say that an end is trivial if 
fr(E) is a single point. 

Lemma 3.5 If E is a non-trivial isolated end of an open subset U of S then 
there is a manifold compactihcation of E by a circle C satisfying the following 
property. If F is any orientation preserving homeomorphism of U U fr(E) 
and if U is F -invariant then F\u extends to a homeomorphism f of U U C . 
Moreover, if G is another orientation preserving homeomorphism of UL) fr(E) 
and if g is the extension of G\u over U U C then: 

1) if F is isotopic to G relative to fr(E) then f is isotopic to g relative to 

C . In particular, if F is isotopic to the identity relative to fr(E) then f 
is isotopic to the identity relative to C . 

2) fg is the extension of (FG)\u . 

If E is a trivial end with fr(E) = {x} and F and G are local diffeomorphisms 
on a neighborhood of x, there is a manifold compactihcation of E by a circle 
C and extensions f, g to homeomorphisms of U U C satisfying property 2). 

Proof Assume at first that E is non-trivial. The existence of C and / is a 
consequence of the theory of prime ends (see [23] for a good modern exposition). 
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Since C is a boundary component of U U C, the extension / is the unique 
extension of F\u over C . Property (2) follows immediately. 

If F and G are as (1), then FG^ 1 is isotopic to the identity relative to fr(E). 
It suffices to show that fg^ 1 is isotopic to the identity relative to C since 
precomposing with g then gives the desired isotopy of / to g. We may therefore 
assume that G is the identity. 

Given an isotopy H t of F to the identity relative to fr(E) , extend H t \u by the 
identity on C to define h t : UUC — > UUC. It suffices to show that h t (x) varies 
continuously in both t and x. This is clear if x € U so suppose that x G C . 
Choose a disk neighborhood system {W{\ for x. It suffices to show that for all 
i and t there exists j so that h s (Wj C\U) C WiC\U for all s sufficiently close to 
t . The frontier of Wj is an arc <jj that intersects C exactly in its endpoints. It 
suffices to show that h s (<jj) is disjoint from <jj and lies in the same component 
of U \ Gi as does <tj . 

The system {W{\ can be chosen with three important properties (see [23] for 
details.) First, the interior of <jj, thought of as an open arc in U, is the interior 
of a closed arc ^ in (7U fr(E) with endpoints in fr(E) . Second, the paths o^s 
are mutually disjoint and converge to a point z G fr(E). The third property 
is that for all i and t, the interior of Ht(a' i ) , thought of as an open arc in U is 
the interior of a closed arc in U U C with the same endpoints as U{ ; this closed 
arc is by definition h t (oi) . 

Fix i and t. Since H t {z) = z, there exists j > i such that H t {u'^) n cr^ = 0. 
By compactness of the closed arcs, H s (a'j) (~l cr^ = for all s sufficiently close 
to t. Since h s (aj) and <jj have the same endpoints, h s (o~j) lies in the same 
component of U \ &i as does <tj . This completes the proof in the case that E 
is non-trivial. 

In the case that E is trivial and F, G are local diffeomorphisms we can construct 
C by blowing up the point x to obtain C and the continuous extensions to 
UUC. The blowing up construction is functorial so property 2) is satisfied. □ 

Proof of Proposition 3.1 We may assume without loss that Fix(F) ^ 0. 
Since G and H commute with F, Fix(F) is G-invariant and iJ-invariant. Let 
(j) be a canonical form for F with minimal reducing set R and let Xi,Si and 
Mi be as in Theorem 2.1. It suffices to show that if Xi is finite then is 
not pseudo-Anosov and that R = 0. These properties are unchanged if F is 
replaced by an iterate so there is no loss in replacing G , H and F by G n , H n 
and F n for some n > 0. Lemma 6.2 of [16] implies that G and permute 



Geometry & Topology, Volume 7 (2003) 



764 



John Franks and Michael Handel 



the elements of R up to isotopy relative to Fix(F). We may therefore assume 
that G and H fix the elements of R up to isotopy relative to Fix(F). 

We first rule out the possibility that some 4>\m % is pseudo-Anosov. For each 
Mi C M there are well defined elements < i^M, >, < G\M t > an d < H\mi > 
in the mapping class group of Mj defined by 'restricting' F, G and H to Mj . 
For concreteness we give the argument for G. Since G preserves the isotopy 
class of each element of R, G preserves the isotopy class of each component of 
dMi and G ~ G± where Gi(Mj) = Mj. If G2 is another such homeomorphism 
then, since G\ ~ G2, Gi|Mj — G^Mi are isotopic as homeomorphisms of Mj. 
Define < G|m< > to be the isotopy class determined by Gi|m;- 

The isotopy classes < F\m { >,< G|Mj > and < H\m { > are determined by 
the actions of F, G and H on isotopy classes of simple closed curves in Mj . 
Thus < F\Mi >= [< G\m{ >< H\m { >] commutes with both < G\m, > and 
< H\Mi > for all i. 

If some < F\m { >=< 4>\Mi > is pseudo-Anosov with expanding lamination 
A, then < G\Mi > and < H\m { > are contained in the stabilizer of A. But 
this stabilizer is virtually Abelian (see, for example, Lemma 2.3 of [19]) in 
contradiction to the fact that < F\mi > m = [< G\mi > m < H\Mi > m ] is non- 
trivial for all m. This completes the proof that is not pseudo-Anosov. 
Thus <fi is the identity on the complement of the annuli Aj . 

If R ^ choose 7 G R, let U be the component of S \ Fix(F) that contains 
7 and let f,g and h be F\u,G\u and respectively. There are three 

cases to consider. The first is that U is an open annulus. By Lemma 3.5 
there is a compactification of U to a closed annulus A and homeomorphisms 
f,g,h,<f>: A — > A that respectively extend f,g,h,(f>\U: U — > C/ and that satisfy 

(1) / commutes with both <? and /i. 




By hypothesis, 



(4) 



is isotopic rel <9A to a non-trivial Dehn twist. 



so 



(5) 



if a is an arc with endpoints on distinct components of dA, then /(d) is 
not isotopic rel endpoints to a. 
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Property (5) contradicts Lemma 3.4 and so completes the proof in this first 
case. 

We may now assume that U has negative Euler characteristic and hence sup- 
ports a complete hyperbolic structure. The second case is that 7 is not pe- 
ripheral in U. There is no loss in assuming that 7 is a geodesic. Choose a 
lift 7 C H to the universal cover of U and let T: H U Soo — ► H U Soo be 
the extended indivisible covering translation that preserves 7. Choose lifts 
g,h: HU S^o — > H U of g, fa that commute with T and so fix the endpoints 
T ± of 7 in Soo. Then / = [g, h] is a lift of / that commutes with T and there 
is a lift (f> of 4>\u that is equivariantly isotopic to /. Let A be the annulus 
obtained as the quotient space of H U (Soo \ T^) by the action of T, and let 
<j),f,g,h: A — > A be the homeomorphisms induced by (f>,f,g and /i. Items 
(2) and (3) above are immediate. Since /<? and gf project to the same home- 
omorphism of U and commute with T, they differ by an iterate of T. Thus 
fd = 9f- The symmetric argument shows that / commutes with h so (1) is 
satisfied. In this case the fixed point set of intersects each component of dA 
in a Cantor set so we must replace (4) with a weaker property (see the proof of 
Proposition 7.1 of [16]) for details): 

(4') (f> is isotopic to a non-trivial Dehn twist relative to a closed set that 
intersects both components of dA. 

Property (5) follows from (4') so the proof concludes as in the previous case. 

The last case is that 7 is peripheral in U and is a minor variation on the sec- 
ond case. By Lemma 3.5, the end of U corresponding to 7 can be compactified 
by a circle and the homeomorphisms f,g,h,cj)\u can be extended to homeo- 
morphisms of the resulting space U* . There is a hyperbolic structure on U* 
in which 7 is isotopic to a peripheral geodesic 7* . The universal cover U* is 
naturally identified with the interior of the convex hull in H of a Cantor set 
C C Soo and is compactifed by a circle consisting of the union of C with the full 
pre-image of 7*. (See for example page 175 of [20].) The proof now proceeds 
exactly as in the second case using U* and its circle compactification in place 
of H and Soo. □ 

Lemma 3.6 Suppose G,H: S — > S are area preserving, orientation preserv- 
ing diffeomorphisms that are isotopic to the identity and that commute with 
their commutator F = [G,H]. Then except in one case all components of 
S \ Fix(i ? ) iiave negative Euler characteristic. The one exception is the case 
that S = S 2 and Fix(F) consists of exactly two points. In this case all compo- 
nents of S\ Fix(F 2 ) have negative Euler characteristic. 
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Proof Let U be a component of S \ Fix(F) . By [4] it is F-invariant. The 
Poincare recurrence theorem and the Brouwer plane translation theorem imply 
that every area preserving homeomorphism of the open disk has a fixed point; 
thus U can not be an open disk. If S = T 2 , then the mean rotation vector of 
F is zero since F is a commutator; by [5], F has fixed points and U ^ T 2 . We 
are left only with the case that U is an open annulus. By Lemma 3.5 we can 
compactify U to a closed annulus A and extend F to A continuously. Suppose 
first that we are not in the exceptional case that S = S 2 and Fix(F) consists 
of exactly two points. Then by Lemma 3.5 the map F is the identity on at 
least one component of <9A. 

Because G and H are area preserving and preserve Fix(F) there exist k, I > 
such that G k {U) = U and if' (17) = U. By Lemma 3.5 we can extend G k ,H l 
to A and by doubling k and I if necessary we may assume G k , H l preserve the 
boundary components of A. 

Then F kl = [G k ,H l ] by Remark 3.2. Let A be the universal covering space of 
A and let u, v : A — > A be lifts of G k and H l respectively. Then w = [u, v] 
is a lift of F kl . Let F be a lift of F which is the identity on one boundary 
component of A. Then F kl is also a lift of F kl . By Lemma 3.4 the map w 
has fixed points in both boundary components of A. It follows that w and F kl 
have a common fixed point and hence they are equal since they are both lifts 
of the same map. 

But the mean rotation of the lift w is zero since it is a commutator. It follows 
from Theorem 2.1 of [15], that it has an interior fixed point. This implies that F 
has an interior periodic point and hence an interior fixed point by the Brouwer 
plane translation theorem. This, in turn, implies F has a fixed point in U 
which is a contradiction. 

We are left with the single exceptional case that S = S 2 and Fix(i ? ) = {p, q} 
so U is the open annulus S 2 \ {p, q}. By Lemma 3.5 we can compactify U to 
form an annulus A and extend F, G and H to orientation preserving, area pre- 
serving homeomorphisms of A. Then G 2 and H 2 must preserve the boundary 
components of A. Suppose first that in addition one of G and H (say G for 
definiteness) preserves the boundary components of A. If u, v are lifts of G 
and H 2 respectively to A then [u,v] has mean rotation zero and is a lift of 
F 2 = [G, H 2 ]. Again applying Theorem 2.1 of [15] we conclude that F 2 has a 
fixed point in the interior of U . 

We want now to show this is also true in the case that both G and H switch 
the boundary components of A. In that case GH and HG must preserve the 
boundary components of A. Let g and h be lifts of G and H respectively to A. 
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They switch the ends of A and hence do not have mean rotation numbers. But 
all elements of the subgroup of the group generated by g and h consisting of 
elements which preserve the ends of of A will have well defined mean rotation 
numbers. This subgroup consists of all elements which can be expressed as 
words of even length in g and h. Let p M (a) denote the mean rotation number 
of an element a in this subgroup and recall p M is a homomorphism. Then 

p,([g 2 , h}) = Pli (g- 2 h-Vh) = P ,((g- 2 h- l g)gh) 

= p„{g- 2 h- l g- l g 2 ) + Plx {gh) = p^g' 1 ) + p^(gh) = 0. 

It follows from Theorem 2.1 of [15] again that [g 2 , h] has a fixed point, but it 
is a lift of F 2 = [G 2 , H] , which must also have a fixed point. 

Hence in all cases Fix(F 2 ) contains at least three points and we can conclude 
from the previous case that if S \ Fix(F 2 ) is non-empty, it has negative Euler 
characteristic. □ 

Lemma 3.7 Suppose F: S — > S is a homeomorphism, that each component 
M of S \ Fix(i ? ) has negative Euler characteristic and that for every n > 0, 
F n is isotopic to the identity rel Fix(F n ). Then Per(F) = Fix(F). 

Proof Let / be the restriction of F to M . We must show Per(/) = 0. Suppose 
to the contrary that x G Per(/). Say it has period p > 1. Choose an arc a 
that initiates at x, terminates at a point y € Fix(i ? ) and is otherwise disjoint 
from Fix(i ? ). By hypothesis, F p {a) is isotopic to a relative to Fix(i ?p ) and 
hence relative to Fix(i ? ) U {x}. Let /: M — > M be the identity lift and let 
do C H be a lift of ao = a \ {y} . The initial endpoint x of d is a lift of x 
and the terminal end of do converges to a point in Soo. The isotopy of F p (a) 
to a relative to Fix(i ? ) U {x} lifts to an isotopy of / p (do) to ao relative to 
S'ooUjx} . But this implies that f p (x) = x in contradiction to the Brouwer plane 
translation theorem and that fact that / is fixed point free. This contradicts 
the assumption that Per(/) / 0. □ 

Proposition 3.8 Suppose G,H : S — > S are area preserving diffeomorphisms 
that are isotopic to the identity and that commute with F = [G, H] . Then 
F 2 = id. If each component M of S\ Fix(F) has negative Euler characteristic 
then F = id. 

Proof We consider the second part first, so we assume M has negative Euler 
characteristic. Then according to Theorem 1.1 of [16] either F has points of 
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arbitrarily high period or is the identity. But according to Corollary 3.3 and 
Lemma 3.7, F has no points of period greater than one. Hence it is the identity. 

For the more general case we need only show that each component of S\Fix(F 2 ) 
has negative Euler characteristic. But this follows from Lemma 3.6. □ 

Example 3.9 Let S 2 be the unit sphere in R 3 and let G: S 2 — > S 2 be 
rotation through the angle 7r around the x-axis. Let H : S 2 -> S 2 be rotation 
through the angle it around an axis in the xy -plane which makes an angle 
of 7r/4 with the x-axis. Both G and H have order 2. One checks easily 
that F = [G, H]: S 2 — > S 2 is rotation around the z-axis through an angle of 
7T. Rotations through angle n around perpendicular axes commute. Hence F 
commutes with G and H and F 2 = id but F ^ id which shows we cannot 
remove the exceptional case in the preceding proposition. The group generated 
by G and H is the dihedral group. 

We are now prepared to prove our main result. 

Theorem 1.2 Suppose Q is an almost simple group containing a subgroup 
isomorphic to the three-dimensional integer Heisenberg group. If S is a closed 
oriented surface then every homomorphism <p: Q — > DifF CJ (5) factors through 
a finite group. 

Proof As shown in the introduction we may assume that the action is by dif- 
feomorphisms isotopic to the identity. Since Q contains a subgroup isomorphic 
to the three-dimensional integer Heisenberg group there are elements G and H 
such that F = [G,H] has infinite order in Q and commutes with G and H. 
By Proposition 3.8 4>(F 2 ) is the identity. It follows that K = ker(<f>) has infi- 
nite order. Since Q is almost simple K C Q has finite index. Hence cf> factors 
through the finite group Q/K. □ 

4 Nilpotent groups 

Suppose Q is a finitely generated group and inductively define Qi for i > by 
Go = G, Gi = [G, Gi-i) := the group generated by {[g, h]\g eQ, he Gi-i}- The 
group G is called nilpotent if for some n, Gn = {e}. If n is the smallest integer 
such that G n = {e} then G said to be of nilpotent length n. G is Abelian, if 
and only if its nilpotent length is 1. 
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Theorem 4.1 Suppose that Q is a finitely generated nilpotent subgroup of 
DiB u (S) . If S / S 2 then Q is Abelian; if S = S 2 then Q has an Abelian 
subgroup of index two. 

Proof Suppose first that S / S 2 . HQ has nilpotent length n > 1 then 
Gn-i is non-trivial and its elements commute with all elements of Q. Since 
Gn-i is generated by commutators there is a non-trivial element F € G n ~i and 
elements G,H GG such that F = [G,H]. But Lemma 3.6 and Proposition 3.8 
then assert that F = id contradicting the assumption that n > 1. 

In case S = S 2 we again assume Q has nilpotent length n > 1 and hence 
that there is a non-trivial element F = [G,H] £ G which commutes with all 
elements of G- If Fix(F) has at least three elements then again by Lemma 3.6 
and Proposition 3.8 F = id contradicting the assumption that n > 1. Hence 
we are reduced to the case Fix(F) consists of exactly two points, {p, q}. 

Let G' be subgroup of G consisting of all elements which fix both p and q. 
Since every element of G commutes with F every element either fixes the two 
points p and q or switches them. Hence G' has index two in Q and is nilpotent. 

We will prove that G' is Abelian. If not then then there is a non-trivial element 
Fq commuting will all of G' and elements Go, Hq € G' such that Fo = [Go, Ho]. 
Blow up the two points p and q for the three diffeomorphisms Fq , Go and Hq . 
Since Fo = [Go, Ho] the mean rotation number of Fo is zero. It follows from 
Theorem 2.1 of [15] that Fo has an interior fixed point. It then follows by the 
argument above that Fo = id which is a contradiction. □ 

Example 4.2 One cannot replace ~DiS u (S)o with Diff^S) in the preced- 
ing theorem. For example, there is an action of the three-dimensional integer 
Heisenberg group on the two dimensional torus T 2 by area preserving diffeomor- 
phisms. We will define the action first on the universal cover M 2 . Choose an ir- 
rational number a and define G, H by (x, y) t— > (x+a, y) and (x, y) i— > (x, x+y) 
respectively. The commutator F of G and H satisfies (x,y) i-> (x,y+a) and so 
commutes with both G and H . The maps G and H descend to area preserving 
diffeomorphisms G, H : T 2 — > T 2 that commute with their commutator. It is 
easy to check that G and H generate a subgroup of Diff w (5) that is isomorphic 
to the three-dimensional integer Heisenberg group. The map H is a Dehn twist 
and so is not contained in Diff^S^o . 

A group G is metabelian if there is a homomorphism to an abelian group whose 
kernel is abelian. 
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Corollary 4.3 Any finitely generated nilpotent subgroup Q of DifF^(S') has 
a finite index metabelian subgroup. 

Proof If S = S 2 , then Q has an index two subgroup that is contained in 
Diff w (S')o so Theorem 4.1 implies that Q is virtually abelian. For S / S 2 
let Diff^S) -> MCG(S) be the natural map. By [2], <$>(Q) is virtually 
abelian. Thus Q has a finite index subgroup Qq whose image <&(Go) is abelian. 
Theorem 4.1 implies that the kernel of $|g i s abelian. □ 
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